ci: Extract test names for flaky test issues#20298
Conversation
size-limit report 📦
|
nicohrubec
changed the title
There was a problem hiding this comment.
I was a bit hesitant about approving this because it seems like this could get more noisier than at the moment. Though then again, it much more aligns with how we'd manually open an issue for flaky CI. I think similarly to the previous iterations we can just give it a try and adjust/revert if it gets too noisy.
One note: wdyt about extracting this script into /srcipts/*? Reading JS in a yml file is quite painful IMHO 😅
| @@ -1,5 +1,5 @@ | |||
| --- | |||
| title: '[Flaky CI]: {{ env.JOB_NAME }}' | |||
| title: '[Flaky CI]: {{ env.JOB_NAME }} - {{ env.TEST_NAME }}' | |||
There was a problem hiding this comment.
This means issue titles could get quite long but I guess it's somewhat necessary when we need an automated way to set the title to something more specific than the job name. Easy to do manually, hard for a machine :D anyway, we can follow up on this. not a blocker for now
There was a problem hiding this comment.
I was thinking we could ask an LLM but that would break deduplication, i.e. we would need to put in a deterministic ruleset else this won't work properly. I don't wanna over engineer this, but we could definitely think about putting in some simple heuristics if we can think of anything that makes sense! I'll merge as is for now and maybe we can revisit later
chargome
left a comment
chargome
left a comment
There was a problem hiding this comment.
+1 for moving this into a dedicated script
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit dc939a7. Configure here.
| describe('Session', () => { | ||
| // TEMPORARY: Guaranteed failure to verify issue creation in CI | ||
| it('TEMPORARY - should fail to test annotation extraction', () => { | ||
| expect(true).toBe(false); |
There was a problem hiding this comment.
m: I think this should go away before merging right?
There was a problem hiding this comment.
yes doing a last test run 😄
This reverts commit dc939a7.
4 participants
Extract the exact failing test name from GitHub check annotations (via Vitest github-actions and Playwright github reporters) and include it in the issue title. This allows us to change issue deduplication from per job to per test. Previously, if test A flaked and created an issue for "Node 18 Integration Tests", a later flake of test B for the same job would be skipped; now each failing test gets its own issue.
Example issue